In some VR systems, a user interacts with any number of virtual objects in a virtual environment using a head-mounted display (HMD) and a hand-held controller. In such VR systems, the user can interact with the objects via the HMD using the hand-held controller. There may be challenges in tracking the hand-held controller using only three degrees of freedom (3DOF) tracking (e.g., orientation tracking).